{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kxu1Gfhx1pHg"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_generative_ai/blob/main/t81_559_class_11_4_eval.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZbNbAV281pHh"
   },
   "source": [
    "# T81-559: Applications of Generative Artificial Intelligence\n",
    "**Module 11: Finetuning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HwnvSYEQ1pHi"
   },
   "source": [
    "# Module 11 Material\n",
    "\n",
    "Module 11: Finetuning\n",
    "\n",
    "* Part 11.1: Understanding Finetuning [[Video]](https://www.youtube.com/watch?v=fflySydZABM) [[Notebook]](t81_559_class_11_1_finetune.ipynb)\n",
    "* Part 11.2: Finetuning from the Dashboard [[Video]](https://www.youtube.com/watch?v=RIJj1QLk-V4) [[Notebook]](t81_559_class_11_2_dashboard.ipynb)\n",
    "* Part 11.3: Finetuning from Code [[Video]](https://www.youtube.com/watch?v=29tUrxrneOs) [[Notebook]](t81_559_class_11_3_code.ipynb)\n",
    "* **Part 11.4: Evaluating your Model** [[Video]](https://www.youtube.com/watch?v=MrwFSG4PWUY) [[Notebook]](t81_559_class_11_4_eval.ipynb)\n",
    "* Part 11.5: Finetuning for Text to Image [[Video]](https://www.youtube.com/watch?v=G_FYFSzkB5Y&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_559_class_11_5_image.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v2MPPX0c1pHi"
   },
   "source": [
    "# Part 11.4: Evaluating your Model\n",
    "\n",
    "## How are Large Language Models Evaluated\n",
    "\n",
    "Before implementing strategies to enhance your fine-tuned language model, it's crucial to understand how OpenAI evaluates these models during the fine-tuning process. This understanding allows you to interpret the metrics provided by the API effectively and make informed decisions to improve your model's performance.\n",
    "\n",
    "### Understanding Training Metrics\n",
    "The primary metrics used to evaluate a model during fine-tuning are the training loss and validation loss. These metrics are calculated using the cross-entropy loss function, a standard method for measuring the difference between the predicted probabilities and the actual distribution of the target data in classification tasks.\n",
    "\n",
    "### Cross-Entropy Loss\n",
    "Cross-entropy loss quantifies the performance of a classification model whose output is a probability value between 0 and 1. In the context of language models, it measures how well the model predicts the next word in a sequence.\n",
    "\n",
    "The cross-entropy loss:\n",
    "\n",
    "\n",
    "$L = -\\sum_{i=1}^{N} y_i \\log(p_i)$\n",
    "\n",
    "where:\n",
    "\n",
    "* $𝑁$ is the number of possible classes (in language models, the vocabulary size).\n",
    "* $y_i$ is the true distribution (1 for the correct word and 0 for others).\n",
    "* $p_i$ is the predicted probability for class \n",
    "\n",
    "For a dataset, the average cross-entropy loss over all predictions provides the training loss or validation loss.\n",
    "\n",
    "### Training Loss\n",
    "The training loss represents the average cross-entropy loss calculated over the training dataset. It reflects how well the model is learning the training data. A decreasing training loss over epochs indicates that the model is effectively capturing patterns within the training data.\n",
    "\n",
    "### Validation Loss\n",
    "The validation loss is computed similarly but over a separate validation dataset not seen by the model during training. It serves as an indicator of the model's ability to generalize to new, unseen data. A low validation loss suggests that the model can effectively apply learned patterns to unfamiliar inputs, not just memorize the training data.\n",
    "\n",
    "### Interpreting Loss Curves\n",
    "Plotting the training and validation loss against epochs can help visualize the model's performance:\n",
    "\n",
    "* **Convergence:** If both losses decrease and eventually stabilize, the model is likely learning effectively.\n",
    "* **Overfitting:** If the training loss continues to decrease while the validation loss starts to increase, the model may be overfitting—memorizing the training data without generalizing well.\n",
    "By analyzing these trends, you can decide whether to adjust training parameters, modify your dataset, or implement techniques like early stopping.\n",
    "\n",
    "### Perplexity as an Evaluation Metric\n",
    "Perplexity is another metric derived from cross-entropy loss, commonly used in language modeling to evaluate how well a probability model predicts a sample.\n",
    "\n",
    "$P=e^L$\n",
    "\n",
    "Where $L$ is the cross-entropy loss. Lower perplexity values indicate better predictive performance, as the model is less \"perplexed\" by the data.\n",
    "\n",
    "### Utilizing the Validation Dataset\n",
    "Incorporating a validation dataset is essential for unbiased evaluation:\n",
    "\n",
    "* **Separate Data:** The validation dataset should be distinct from the training data to provide an accurate assessment of the model's generalization capabilities.\n",
    "* **Loss Calculation:** The validation loss is computed using cross-entropy loss over the validation dataset after each epoch.\n",
    "By comparing training and validation loss, you can detect issues like overfitting and adjust your training strategy accordingly.\n",
    "\n",
    "Accessing Detailed Training Metrics\n",
    "OpenAI's API offers detailed logs and metrics during the fine-tuning process:\n",
    "\n",
    "\n",
    "## How can you Improve Results when Finetuning a Large Language Model\n",
    "\n",
    "Fine-tuning large language models, such as those provided by OpenAI, allows developers to tailor these powerful tools to specific tasks and domains. Achieving optimal results requires more than just running the fine-tuning process; it involves a strategic approach to data preparation, parameter adjustment, and iterative improvement. This chapter explores various methods to enhance the outcomes of fine-tuning an OpenAI language model using the API.\n",
    "\n",
    "### High-Quality Training Data\n",
    "The cornerstone of effective fine-tuning is the quality of the training data.\n",
    "\n",
    "* **Relevance:** Ensure that your dataset is closely aligned with the tasks or topics you want the model to handle. For instance, if you're developing a model for medical diagnoses, include medical case studies and terminologies.\n",
    "\n",
    "* **Clarity and Consistency:** Use clear, precise language to prevent ambiguity. Maintain a consistent style, tone, and formatting throughout the dataset to help the model learn the desired patterns effectively.\n",
    "\n",
    "### Sufficient Quantity of Data\n",
    "While quality is paramount, the amount of data also influences the model's performance.\n",
    "\n",
    "* **Comprehensive Examples:** A larger dataset provides the model with more patterns to learn from, improving its ability to generalize to new inputs.\n",
    "\n",
    "* **Balanced Dataset:** Include a diverse range of examples to cover different scenarios, but avoid unnecessary repetition that could lead to overfitting.\n",
    "\n",
    "### Optimizing Data Formatting\n",
    "Properly structured data guides the model during training.\n",
    "\n",
    "* **Use JSONL Format:** OpenAI recommends the JSON Lines (JSONL) format, where each line is a JSON object containing \"prompt\" and \"completion\" keys.\n",
    "\n",
    "Structured Prompts and Completions: Clearly define the input (prompt) and the expected output (completion). For example:\n",
    "\n",
    "```\n",
    "{\n",
    "  \"messages\": [\n",
    "    {\"role\": \"system\", \"content\": \"You are a language translation assistant.\"},\n",
    "    {\"role\": \"user\", \"content\": \"Translate to French: Hello, how are you?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"Bonjour, comment ça va?\"}\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "Include Metadata When Necessary: Additional information, such as labels or context identifiers, can be embedded in the prompts to provide the model with more context.\n",
    "\n",
    "### Crafting Instruction-Based Prompts\n",
    "Directing the model with well-designed prompts can enhance its responses.\n",
    "\n",
    "* **Explicit Instructions:** Begin prompts with clear instructions or questions. For example, \"Explain the significance of photosynthesis in plants.\"\n",
    "\n",
    "* **Consistent Prompt Format:** Maintain a uniform structure in prompts to help the model recognize and replicate the desired response patterns.\n",
    "\n",
    "### Adjusting Training Parameters\n",
    "Fine-tuning parameters significantly affect the model's learning process.\n",
    "\n",
    "* **Epochs:** Experiment with the number of epochs—the number of times the model passes through the entire training dataset. Too few epochs may lead to underfitting, while too many can cause overfitting.\n",
    "\n",
    "* **Batch Size:** Adjust the batch size, which is the number of training examples used in one iteration. A larger batch size can speed up training but may require more computational resources.\n",
    "\n",
    "* **Learning Rate:** The learning rate controls how much the model adjusts its weights with each update. A suitable learning rate ensures stable convergence.\n",
    "\n",
    "### Data Augmentation Techniques\n",
    "Enhancing your dataset through augmentation can improve model robustness.\n",
    "\n",
    "* **Paraphrasing:** Rephrase sentences to provide the model with varied inputs that have the same meaning.\n",
    "\n",
    "* **Synonyms and Antonyms:** Replace words with synonyms to diversify vocabulary exposure.\n",
    "\n",
    "* **Noise Introduction:** Intentionally introduce minor errors or variations to help the model handle imperfect inputs.\n",
    "\n",
    "### Regular Evaluation and Iteration\n",
    "Ongoing assessment allows for continuous improvement.\n",
    "\n",
    "* **Validation Set:** Reserve a portion of your data as a validation set to evaluate the model's performance objectively.\n",
    "\n",
    "* **Performance Metrics:** Monitor metrics such as accuracy, loss, and perplexity to gauge improvement.\n",
    "\n",
    "* **Iterative Refinement:** Use insights from evaluations to refine your training data and adjust parameters in subsequent training rounds.\n",
    "\n",
    "## Leveraging Advanced API Features\n",
    "OpenAI's API provides options to fine-tune model outputs during inference.\n",
    "\n",
    "* **Temperature Setting:** Controls the randomness of the output. Lower values make responses more deterministic, while higher values increase creativity.\n",
    "\n",
    "* **Top-p (Nucleus Sampling):** Adjusts the cumulative probability threshold for token selection, balancing the trade-off between diversity and focus.\n",
    "\n",
    "* **Max Tokens:** Limits the length of the generated output to prevent excessively long responses.\n",
    "\n",
    "### Implementing Early Stopping\n",
    "Prevent overfitting by stopping training at the optimal point.\n",
    "\n",
    "* **Monitor Loss Trends:** Observe the training and validation loss. If validation loss starts increasing while training loss decreases, overfitting may be occurring.\n",
    "\n",
    "* **Set Patience Levels:** Define a number of epochs with no improvement after which training will stop.\n",
    "\n",
    "Multiple Fine-Tuning Rounds\n",
    "Sequential fine-tuning can progressively enhance model performance.\n",
    "\n",
    "* **Initial Broad Training:** Start with a general dataset to teach the model basic patterns.\n",
    "\n",
    "* **Focused Refinement:** In subsequent rounds, use more specific data to hone the model's performance on particular tasks.\n",
    "\n",
    "### Incorporating Negative Examples\n",
    "Teaching the model what not to do can be as important as teaching it what to do.\n",
    "\n",
    "* **Incorrect Examples:** Include prompts that lead to incorrect completions along with corrections.\n",
    "\n",
    "* **Penalty Mechanisms:** While not directly supported, structuring data to discourage certain outputs can guide the model away from undesired responses.\n",
    "\n",
    "### Ensuring Dataset Diversity\n",
    "A diverse dataset helps the model handle a wide range of inputs.\n",
    "\n",
    "* **Varied Topics:** Incorporate content from different subject areas.\n",
    "\n",
    "* **Stylistic Variation:** Use examples with different writing styles, tones, and formats.\n",
    "\n",
    "### Monitoring Training Metrics\n",
    "Keep an eye on the model's learning process.\n",
    "\n",
    "* **Loss Curves:** Plotting training and validation loss over epochs can reveal learning patterns.\n",
    "\n",
    "* **Accuracy Metrics:** Where applicable, track how often the model produces correct responses.\n",
    "\n",
    "### Effective Prompt Engineering\n",
    "Designing prompts that elicit the desired response is an art.\n",
    "\n",
    "* **Use Placeholders:** Employ variables in prompts to generalize patterns. For example, \"Calculate the sum of {number1} and {number2}.\"\n",
    "\n",
    "* **Directive Language:** Start prompts with verbs like \"Explain,\" \"Describe,\" or \"Summarize\" to guide the model.\n",
    "\n",
    "### Incorporating User Feedback\n",
    "Real-world usage provides valuable insights.\n",
    "\n",
    "* **Collect Feedback:** Gather responses from users to identify strengths and weaknesses.\n",
    "\n",
    "* **Update Training Data:** Use this feedback to adjust your dataset, adding new examples or correcting existing ones.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
